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Abstract. In this paper we consider the analytic continuation 
of the weighted Bergman spaces on the Lie ball 

9 = SO{2,n)/S{0{2) x 0(n)) 

and the corresponding holomorphic unitary (projective) represen- 
tations of SO(2,n) on these spaces. These representations are 
known to be irreducible. Our aim is to decompose them under the 
subgroup SO(l,n) which acts as the isometry group of a totally 
real submanifold % of 9. We give a proof of a general decomposi- 
tion theorem for certain unitary representations of semisimple Lie 
groups. In the particular case we are concerned with, we find an 
explicit formula for the Plancherel measure of the decomposition 
as the orthogonalising measure for certain hypergeometric poly- 
nomials. Moreover, we construct an explicit generalised Fourier 
transform that plays the role of the intertwining operator for the 
decomposition. We prove an inversion formula and a Plancherel 
formula for this transform. Finally we construct explicit realisa- 
tions of the discrete part appearing in the decomposition and also 
for the minimal representation in this family. 



Introduction 

One of the main problems in the representation theory of Lie groups 
and harmonic analysis on Lie groups is to decompose some interest- 
ing representations of a Lie group G under a subgroup H C G. This 
decomposition is also called the branching rule. Among other things, 
this has led to the discovery of new interesting representations. An 
exposition of the general theory for compact connected Lie groups, 
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including the classical results for U{n) and SO(n) (by Weyl and Mur- 
naghan respectively), can be found in [T2] . 

Since the work by R. Howe [7] and M. Kashiwara and M. Vergne (cf 
[TO]), it has turned out to be fruitful to study the branching of singular 
and minimal holomorphic representations of a Lie group acting on 
a function space of holomorphic functions on a bounded symmetric 
domain. In |9j, Jakobsen and Vergne study the restriction of the tensor 
product of two holomorphic representations to the diagonal subgroup. 

In this paper we will study the branching of the analytic con- 
tinuation of the scalar holomorphic discrete series of SO(2,n) un- 
der the subgroup H = SOo(l,n). The subgroup H here is realised 
as the isometry group of a totally real submanifold of the Lie ball 
SO(2,n)/S(0(2) x 0{n)). The branching for a general Lie group G 
of Hermitian type under a symmetric subgroup H has been studied 
recently by Neretin ([19], pj). Zhang ([28], [30], [29]) and by van Dijk 
and Pevzner [25] . In [T4], Kobayashi and 0rsted studied the branching 
for some minimal representations. The branching rule for regular pa- 
rameter and for some minimal representations is now well understood. 
However, the problem of finding the branching rule for non-discrete, 
non-regular parameter is a difficult one, and there is still no complete 
theory for the general case. 

We find the branching rule for arbitrary scalar parameter v in the 
Wallach set of SO(2, n). It turns out that for small parameters v 
there appears a discrete part in the decomposition. We discover here 
an intertwining operator realising the corresponding representation. It 
should be mentioned that for large parameter (in this case v > n — 1) 
the corresponding branching problem has been solved by Zhang in [28] 
for arbitrary bounded symmetric domains. 

The paper is organised as follows. In Section 1 we describe the ge- 
ometry of the Lie ball. In Section 2 we recall some facts about general 
bounded symmetric domains and Jordan triple systems. In Section 3 
we establish some facts about the real part of the Lie ball. In Section 
4 we consider a family of function spaces and corresponding unitary 
representations. Section 5 is devoted to branching theorems and to 
finding the Plancherel measure. In Sections 6 and 7 we find realisa- 
tions of the representations corresponding to the discrete part in the 
decomposition and to the minimal point in the Wallach set respec- 
tively. 
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1. The Lie ball as a symmetric space 
SO (2,n)/SO(2) x SO(n) 

In this paper we study representations on function spaces on the 
domain 

(1) @ = {z e C n |l - 2(z, z) + \zz'\ 2 > 0, |s| < 1}. 

We will only be concerned with the case n > 2. (If n — 1 it is the 
unit disk, U, and if n — 2, 3> = U x U). In this section we describe 
3> as the quotient of SO (2,n) by SO (2) x SO(n)) by studying a 
holomorphically equivalent model on which we have a natural group 
action induced by the linear action on a submanifold of a Grassmanian 
manifold. Consider M n+2 = M 2 ©IR n equipped with the non-degenerate 
bilinear form 

(x\y) := x x y x + x 2 y 2 - x 3 y 3 - ... - x n+2 y n +2, 

where the coordinates are with respect to the standard basis ei, • • • , e n+2 
Let SO(2,n) be the group of all linear transformations on IR n+2 that 
preserve this form and have determinant 1, i.e., 

SO(2,n) = {ge GL(2 + n, R)\(gx\gy) = (x\y),x,y G R 2+n ,det g = 1} 

Let n ) denote the set of all two-dimensional subspaces of M 2 © W 1 
on which (•]•) is positive definite. Clearly R 2 © {0} is one of these 
subspaces. It will be the reference point in QX ^ and we will denote it 
by Vq. The group SO(2, n) acts naturally on this set and the action is 
transitive. In fact, the connected component of the identity, SOo(2, n) 
acts transitively. We will let G denote this group. 
We denote by K the stabilizer subroup of Vq, i.e., 



(2) 



K = {ge G\g(V ) = V }. 
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Any element g G G can be identified with a (2 + n) x (2 + n)- matrix 
of the form 



(3) 



A B 
C D 



where A is a 2 x 2-matrix. With this identification, K clearly corre- 
sponds to the matrices 

A 
D 

where A and D are orthogonal 2x2- and n x n-matrices with deermi- 
nant one respectively, i.e., K = SO '(2) x SO(n). The space Gt 2n ) can 
be realised as the unit ball in M n2 (R) with the operator norm. Indeed, 
let V G Gt 2n y If v — Vi + v 2 G V, then t> i = implies that f 2 = 0, 
i.e., the projection v vi is an injective mapping. This means that 
there is a real n x 2 matrix Z with Z*Z < such that 

(4) 7 = {(v® Zv)\v eR 2 }. 

Conversely, if Z G M n2 (M) satisfies Z'Z < I 2l then (jlj) defines an 
element in G% n y 

Using ([3]) to identify g with a matrix and letting V correspond to 
the matrix Z, then clearly 

g V = {(Av + BZv®Cv + DZv)\v G R 2 )} 
= {f © (C + DZ)(A + BZ)~ l v)\v G IR 2 )}. 

In other words, we have a G-action on the set 

M= {Z G M n2 (R)|Z*Z < I 2 } 

given by 



(C + DZ^A + BZ)- 1 . 
This exhibits M as a symmetric space. 

M = G/K. 

Moreover, we identify the matrix Z = (XY) with the vector X + iY 
in C n in order to obtain an almost complex structure on M. With 
respect to this almost complex structure, the action of G is in fact 
holomorphic. Moreover we have the following result by Hua (see [8]). 
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Theorem 1. The mapping 

where zz l = z\ + • • • + z\, is a holomorphic diffeomorphism of the 
bounded domain 

@ = {ze C n |l - 2(z, z) + \zz l \ 2 > 0, \z\ < 1} 

onto M. 

We will call this mapping the Hua transform. It allows us to describe 
f as a symmetric space 

Q) = M = G/K. 

2. Bounded symmetric domains and Jordan pairs 

In this section we review briefly some general theory on bounded 
symmetric domains and Jordan pairs. All proofs are omitted. For a 
more detailed account we refer to Loos ([E]) and to Faraut-Koranyi 

(0)- 

Let V be a bounded open domain in C n and 7i 2 {V) be the Hilbert 
space of all square integrable holomorphic functions on T>, 

H 2 (V) = {/,/ holomorphic on V\ [ \ f{z)\ 2 dm{z) < oo}, 

Jv 

where m is the 2n-dimensional Lebesgue measure. It is a closed sub- 
space of L 2 (T>). For every w 6 X>, the evaluation functional / i— > f{w) 
is continuous, hence 7i 2 {V) has a reproducing kernel K(z,w), holo- 
morphic in z and antiholomorphic in w such that 

f( w ) = / f{z)K(z,w)dm(z). 
Jv 

K(z,w) is called the Bergman kernel. It has the transformation prop- 
erty 

(5) K(<f(z), ip{w)) = J^z^Kiz, w)JJw)~\ 
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for any biholomorphic mapping ip on V with complex Jacobian J v (z) = 
det dip(z). Hereafter biholomorphic mappings will be referred to as au- 
tomorphisms. The formula 

(6) h z (u,v) = d u dvlog K(z,z) 

defines a Hermitian metric, called the Bergman metric. It is invariant 
under automorphisms and its real part is a Riemannian metric on V. 

A bounded domain V is called symmetric if, for each z G V there is 
an involutive automorphism s z with z as an isolated fixed point. Since 
the group of automorphisms, Aut(T>) preserves the Bergman metric, 
s z coincides with the local geodesic symmetry around z. Hence V is 
a Hermitian symmetric space. 

A domain V is called circled (with respect to 0) if G T> and 
e lt z G V for every z6D and real t. 

Every bounded symmetric domain is holomorphically isomorphic 
with a bounded symmetric and circled domain. It is unique up to 
linear isomorphisms. 

From now on V denotes a circled bounded symmetric domain. G is 
the identity component of Aut(V) 7 K is the isotropy group of in G. 
The Lie algebra q will be considered as a Lie algebra of holomorphic 
vector fields on V, i.e., vector fields X on V such that Xf is holomor- 
phic if / is. The symmetry s,z h — z around the origin induces an 
invoulution on G by g i— > sgs^ 1 and, by differentiating, an involution 
Ad(s) of g. We have the Cartan decomposition 

g = i ©P 

into the ±l-eigenspaces. 

For every v G C n , let £ v be the unique vector field in p that takes 
the value v at the origin. Then 

(7) W) =v~ Q(z)v 

where Q(z) : V — > V is a complex linear mapping and Q : V — > 
Hom(V, V) is a homogeneous quadratic polynomial. Hence Q(x, z) = 
Q(x + z) — Q(x) — Q(z) : V — > V is bilinear and symmetric in x and 
z. For x,y,z G V, we define 

(8) {xp} = D(x, y)z = Q(x, z)y 

Thus {xyz} is complex bilinear and symmetric in x and z and complex 
antilinear in y, and D(x,y) is the endomorphism z i— > {xp} of V. 
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The pair (V, { }) is called a Jordan triple system. This Jordan triple 
system is positive in the sense that if v G V, v ^ and Q(v)v = Xv for 
some A G C, then A is positive. We introduce the endomorphisms 

(9) B(x,y) = I- D(x, y) + Q(x)Q(y) 



of V for x, y G V, where Q(y)x = Q(y)x. We summarise some results 
in the following proposition. 

Proposition 2. I7ie Lie algebra g satisfies the relations 

(10) = - D(u,u) 

(11) = 6« 

/or u,D£y anc? Z G t 

6 J T/ie Bergman kernel k(x,y) ofD is 

(12) m(V)- 1 de^B(x,y)- 1 

c) The Bergman metric at is 

(13) h (u,v) — trD(u,v), 
and at an arbitrary point z G V 

(14) /i z (u, v) = h (B(z, z)~ x u, v) 

d) The triple product { } is given by 

(15) h ({uvw},y) = d u c\d x dy\ogK(z,z)\ z=0 
We define odd powers of an element x G V by 



X L = X: X A = Q( x )x, • • • , x zn+v = Q(x)x 2n ~ 1 . 

An element x G V is said to be tripotent if x 3 = x, i.e., if {xafx} = 
2x. Two tripotents c and e are called orthogonal if D(c,e) = 0. In 
this case D(c,c) and D(e,e) commute and e + c is a tripotent. 

Every x G V can be written uniquely 

x = AiCi H h A n c„, 

where the Cj are pairwise orthogonal nonzero tripotents which are real 
linear combinations of odd powers of x, and the A, satisfy 

< Ai < • • • < A n . 
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This expression for x is called its spectral decomposition and the \ 
the eigenvalues of x. Moreover, the domain V can be realised as the 
unit ball in V with the spectral norm 

||a;|| = max |Aj|, 

where the A, are the eigenvalues of x, i.e., 

v = {xe V\\\x\\ < l}. 

Let f(t) be an odd complex valued function of the real variable t, 
defined for \t\ < p. For every x G V with |x| < p we define f(x) G V 

by 

(16) f(x) = f(X 1 )c 1 + --- + f(X n )c n , 

where x = X±ci + • • • + X n c n is the spectral resolution of x. This func- 
tional calculus is used in expressing the action on V of the elements 
exp£y in G: 

(17) exp£v(z) = u + B(u,u) 1/2 B(z, -u)~ l (z + Q(z)u) 
and 

(18) d(expZ v )(z) = B(u,u) 1/2 B(z,-u)-\ 
where u = tanhf , for v G C n and z G V. 

3. The real part of the Lie ball 
We consider the non-degenerate quadratic form 

(19) q(z) =z 2 + --- + z 2 n 

on V = C™. In the following we will often denote q(z,w) by (z,w). 
Defining Q(x)y = q(x, y)x — q(x)y, where q(x, y) = q(x + y) — q(x) — 
q(y), we get a Jordan triple system. The Lie ball &> — {z G C n |l — 
2(z, z) + | zz f | 2 > 0, \z\ < 1} is the open unit ball in this Jordan triple 
system. An easy computation shows the following identity. 

n n n 

D(x, y)z = 2(^2 x kVk)z + 2(^2 z kVk)x - 2(J^ x k z k )y 

k=l k=l k=l 

Recalling that B(x,y) = I — D(x,y) + Q(x)Q(y) . The Bergman kernel 
of is 

(20) K(z, w) = (l- 2(z, w) + (zz t )(ww t ))- n . 
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We will hereafter denote it by h(z, w) n . Consider the real form IR n 
in C n . Observe that 

:= @f)R n 

is the unit ball of M n . On 2£ we have a simple expression for the 
Bergman metric: 

(21) B(x,x) = (1 - \x\ 2 )- 2 I,x G 5C. 

The submanifold 2£ is a totally real form of in the sense that 

T X {X) + %T X {.T) = T x (@), T x {.T)^iT x {%) = {0} 

This implies that every holomorphic function on Q) that vanishes on 
S£ is identically zero. We define the subgroup H as the identity 
component of 

{h G G\h(x) G 3£ if a; G J"} 

We will denote H f| K by L. 

Using the fact that the real form M. n is a sub-triple system of C n , 
one can show that JT is a totally geodesic submanifold of Q) (cf Loos 
[15]). Hence we can describe 3E as a symmetric space 

We now study the image of in the M n2 (M)- model of the Lie ball. 
For computational convenience, we now work with the transposes of 
these matrices. The defining equation of the Hua-transform can be 
written as 

1 f zz* + l i{zz l - 1) \ _ / Z \ 
2\z? + l -i(zz l -1) ) \z J 

In the coordinates (zi, . . . , z n ) of z, this identity takes the form 

(23) z k = -( (zz* + l)x k + i{zz l - l)y k ). 
This gives 

(24) Azz t = {zz t f{X + iY)(X + iY) 1 + 2(XX* + FX*)z^ 

(25) +{X - iY)(X - iYf, 
which is a quadratic equation in zz l with unique solution 

(26) zz l 

2 - (XX* + YY l ) - 2^(1 - XX*)(1 - YY l ) - (FX*) 2 
~ (X + iY)(X + iYY ' 
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From ( 1231) we see that if z is real, then = for all k. On the other 
hand, if Y = 0, then (1261 ) shows that zz* is real and therefore z is 
real by (1231) . Hence the image of the real part C & under the 
Hua-transform is the set 



(27) 



H{SC) = {Z = (X0) \X E M, 



71 1 



\x\<i}, 



since for an element Z = (X0), the condition that Z t Z < I2 is clearly 
equivalent with \X\ < 1. 

Recall that the real rz-dimensional unit ball can be described as a 
symmetric space SOo(l,n)/ SO(n) by a procedure analogous to the 
one in the first section. One first considers all lines in IR 1+n on which 



the quadratic form x\ — x 



_ x is positive definite and identifies 



these lines with all real n x 1-matrices with norm less than one. If we 
write elements g G SO(l,n) as matrices of the form 



(28) 



9 



fa- b - \ 

c D 

V 1 / 



the action is given by 

(29) X ^ (c + DX)(a + bX)- 1 . 

The group 50(1, n) can be embedded into SO(2,n). Indeed, the 
equality 



/ a 
1 

c 



b 


D 



\ 



( a 





1 



c' 



V 


D' 



J \ 



\ 



( aa! + bc' - ah' + bD' -\ 
1 - 

ca' + Dc' cb' + DD' 
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shows that we can define an injective homomorphism 9 : SO(l, n) 
SO (2, n) by 



9 : 



V 



b 

D 



\ 
J 



I 


a 


- 


b 


- \ 







1 - 









c 





D 




\ 








J 



This subgroup acts on H.(3£) as 

(10) i — > ((c + J DX)(a + 6X)- 1 0) 

and the action is transitive. Suppose now that /i G SO(2,n) preserves 
H(3£). Let p = h(0). We can choose a j 6 50o(l,ri) such that 
g(0) = p (here we identify g with 9(g)). Then g _1 /i(0) = and hence 
we can write it in block form as 



9 ~ lh= {o d) 



with D E SO(n). This is an element in 9(SO(l,n)) and hence h E 
9(SO(l,n)). We have now proved the following theorem. 



Theorem 3. The Hua transform H 
diffeomorphically onto 



M maps the real part 2£ 



H(%) = {Z = (10) \X E M nl (R), \X\ < 1} 



(30) 

by x l— *■ i+\ x \ 2 ■ Moreover, the induced group homomorphism h \— > 
TihTi^ 1 is an isomorphism between the groups H and SOo(l,n) 

Remark. The model Ti(S^) of SO (l,n)/ SO(n) is the real part of 
the complex n-dimensional unit ball SU(l,n)/ SU(n) with fractional- 
linear group action. It is therefore equipped with a Riemannian metric 
given by the restriction of the Bergman metric of the complex unit 
ball. If x E H{3P),x ^ 0, we decompose R n = Rx © (Rx^. We let 
v = v x + v x ± be the corresponding decomposition of a tangent vector 
v at x. In this model, the Riemannian metric at x is (cf [21J) 



9x(v,v) 



(1 - \x 



2\2 



+ 







2 


(1 




X 


2 ) 
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We recall from equation ( |2T| ) that if x G X, then the Riemannian 
metric at x is 



h x (v,v) = 



1 


V 


2 


2n(l - 




x\ 


2\2 



The Hua transform thus induces an isometry (up to a constant) of the 
real n-dimensional unit ball equipped with two different Riemannian 
structures. 

3.1. Iwasawa decomposition of h. The Cartan decomposition g = 
t © p induces a decomposition f) = I © q. We let 



be the one-dimensional subspace of q, where e = e% denotes the first 
standard basis vector and the corresponding vector field £ e is defined 



Proposition 4. The Lie algebra rj has rank one, and the roots with 
respect to the abelian subalgebra a of q are {a, —a}, where a(£ e ) = 2. 
The corresponding posive root space is 



Proof. This is known in a general context, but we give here an ele- 
mentary proof. 

Take u and v in M™ and assume that = 0. Then, for any 

x G W 1 we have 



which can only hold for all real x if u = v. 

Thus a is a maximal abelian subalgebra in q. The vector e is a 
maximal tripotent in the Jordan triple system corresponding to @. 
Suppose that [£ e , + I] — cx(£ e )(£ v + I). Identifying the q- and l- 
components yields 



in ©. 



f )x = -D(t>, w)a;. 
A simple calculation shows that this amounts to 

(u, x)v = (v, x)u, 



(31) 
(32) 



D(e,v)-D(v,e) = a(Ql 
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From f l32i ) it follows that le = —a(^ e )v and, thus, applying both sides 
of (l3Ti ) to e gives 

D(e, v)e - D(v, e)e = -a(£, e ) 2 v, 

i.e., 

D(e, e)v — D(e, v)e = a(£ >e ) 2 v ) 

An easy computation gives 

Av — 4(e, v)e = a(£ e ) 2 v. 

Hence e is orthogonal to v and a(£ e ) 2 — 4. The rest follows immedi- 
ately. □ 

We shall fix the positive root a. Elements in are of the form 
Xa and will hereafter be identified with the complex numbers A. In 
particular, the half sum of the positive roots (with multiplicities), p, 
will be identified with the number (n — l)/2. 

3.2. The Cay ley transform. The Cayley transform is a biholomor- 
phic mapping from a bounded symmetric domain onto a Siegel domain. 
We describe it for the domain Q) and use it to express the spherical 
functions on in terms of the spherical functions on the unbounded 
domain. We fix the maximal tripotent e. Then C n equipped with the 
bilininear mapping 

(33) (z, w) t— > z o w = -{zew} 

is a complex Jordan algebra. Observe that since e is a tripotent, it is 
a unity for this multiplication. The Cayley transform is the mapping 
c : C n -> C n defined by 

(34) c(z) = (e + z) o (e - z)~\ 

where (e — z)~ x denotes the inverse of (e — z) with respect to the 
Jordan product. 

Proposition 5. The Cayley transform is given by the formula 

- 1 ~ — 2z ' 
{ ' C[Z} ~ l-2z 1 + (zz t y 6+ 1-2^ + ^)2' 

for z = (z±, z') = zie+z' G @. Moreover, it maps & onto the halfspace 

{( Xl ,...,x n }eR n \xi >0}. 
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Proof. We first find the inverse for an element x. Suppose therefore 
that e = ^{xez} = ^D(x,e)z, i.e., 

e = (x, e)z + (z, e)x — (x, z)e = x\Z + Z\x — (x, z)e 

Identifying coordinates gives 

1 = 2x\Z\ — (x, z) 

= x\z' + Z\x' 

These equations have the solution 

z\ = x±/(x,x) 

z' = —x'/(x,x). 

If we apply this to the expression (e — z)~ l in the definition of c, we 
get 

{6 ~ Z) 1= (l-^) 2 + (^^) e+ (l-^i) 2 + (^^)' 
Now the formula (|35l) follows by an easy computation. Moreover, we 
observe that the inverse transform is given by 

iDH(to-e)o(to-|- e) _1 = —c(—w). 

Hence both c and c _1 preserve R n and therefore 

c{3£) = c(^)p|M n . 

We now determine c( JT). 

From (|15|) we know that (since e is a maximal tripotent) 

(36) c{@) = {u + iv\u e A + ,v G A}, 

where A is the real Jordan algebra 

{zeV\Q(e)z = z} 

and A + is the positive cone {z o z\z e A} in A. By a simple compu- 
tation we see that 

A = Re © Rie 2 © • • ■ © Kie n . 
Since we have the identities 

z + Q(e)z = 2u, 
z — Q(e)z = 2iv 

and 

Q(e)z = 2z~[-z, 
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we get expressions for u and v : 

2u = (z 1 + zl, z 2 - Z2, ■ ■ ■ , z n - ~zt) 

2iv = (zi-zi,z 2 +Z2,...,z n + z^) 
The condition that x = u + iv be in the image of 3£ thus implies that 

u = (xi, 0, ... ,0), 

iv = (0,x 2 , ■ ■ -,x n ). 
Moreover we require that 

u = w o w = 2wiw — (w, w)e, 

for some 

w = cie + c 2 ie 2 H h c n ie n . 

This yields 

(xi, . . . , 0) = (c? H h 4, icic 2 , • • • , iciCn). 

Hence 

c l = 2-1) c 2 = ' ' ' = c n = 0, 

and thus 

u + iv = (cl,x 2 , ■ ■ ■ ,x n ). 
This proves the claim. □ 

Recall the expression for the spherical functions on a symmetric 
space of noncompact type (cf [6] Thm 4.3) 

tp^h) = J e (lX+p)A{lh) dl, 

where A(lh) is the (logarithm) of the A part of Ih in the Iwasawa 
decomposition H = NAL. The integrand in this formula is called 
the Harish- Chandra e- function. For the above Siegel domain it has 
the form e\(w) = (wi) lX+p (cf pi]). Hence we have the following 
corollary. 

Corollary 6. The spherical function ip\ on = H/L is 

where a is the 0{n) -invariant probability measure on S' n-1 . 
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4. A FAMILY OF UNITARY REPRESENTATIONS OF G 

4.1. The function spaces J%,. The Bergman space 7i 2 {&) has the 
reproducing kernel h(z,w)~ n . This means in particular that the func- 
tion h(z,w)~ n is positive definite in the sense that 

m 

^ aiajh(zi, ZjY n > 0, 

for all z±, . . . , z n 6 @ and a±, . . . , a n E C. It has been proved by Wal- 
lach (|26|) and Rossi- Vergne ([2D]) that h(z,w)~ u is positive definite 
precisely when v in the set 

{0,(n-2)/2}U((n-2)/2,oo) 

This set will also be referred as the Wallach set (cf [3]). For v in the 
Wallach set above, h(z,w)~ u is the reproducing kernel of a Hilbert 
space of holomorphic functions on *3l . We will call this space J^, and 
the reproducing kernel K u (z,w). The mapping g i— > 7r v (g), where 

7r^)/(z) = Jg-^fig-h) 

defines a unitary projective representation of G on J^,. Indeed, com- 
parison with the Bergman kernel shows that h(z, w)~ u transforms un- 
der automorphisms according to the rule 

(38) h(gz,gw)- v = J g (z)-$h(z, w)-"jjwj~" . 
Recall that for functions f\ and / 2 of the form 

l m 

fi( z ) = ^2akK u (z,w k ), f 2 (z) = ^2f3 k K u (z,w' k ), 

k=l k=l 

the inner product is defined as 

(39) (fi, f 2 )u = ^2 <XiPjK v (y)i, w'j) 

Equation (1351 implies that 

(40) K v (g- X z,w) = J g -i(z)-ZK u (z,gw)J g -i(w)~". 
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Hence we have the following two equalities 



Kv{g)h{z) 



^a k J g -i{w k ) n K u (z,gw k ) 



k=l 



rn 



Y J hJ 9 -A<y~K u {z,gw' k ). 



k=l 



The unitarity 



Mg)fu K v {g)f-2) v = (A, f 2 ) 



V 



now follows by an application of the transformation rule ( |38l ) in the 
definition ( 1391) . Since functions of the form above are dense in J^,, it 
follows that each ir u (g) is a unitary operator and it is easy to see that 
g | — ► K v (g) is a projective homomorphism of groups. In fact, ir u is an 
irreducible projective representation, cf [2]. 

4.2. Fock-Fischer spaces. It can be shown that for v > (n — 2)/2 all 
holomorphic polynomials are in Jjf u and that polynomials of different 
homogeneous degree are orthogonal. In this context, the spaces 3^ v are 
closely linked with the Fock-Fischer space, J£~, which we will now de- 
scribe. The basis vector e\ is a maximal tripotent which is decomposed 
into minimal tripotents as e± = |(1, i, 0, . . . , 0) + |(l, —i, 0, . . . , 0). (We 
omit the easy computations.) In order to expand the reproducing ker- 
nel K v into a power series consistent with the treatment in [2j, we 
need to introduce a new norm on C n so that the minimal tripotents 
have norm 1, i.e., the Euclidean norm multiplied with \/2. Then 



is an orthonormal basis with respect to this new norm. We write points 
z G Q> as z = Wifi + ■ ■ • + w n f n . For polynomials p(w) = J2 a a aW a , 
we define 



{fi, ■ ■ ■ , fn} '■ 




a n w 



a 



The Fock-Fischer inner product is now defined as 



{p,q)<? = p(d)(q*)\ w=0 , 
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where p(d) is the differential operator ^ tt o a ^, for p as above. The 
Fock-Fischer space, J^, is the completion of the space of polynomials. 
It is easy to see that polynomials of different homogeneous degree 
are orthogonal in Moreover, the representation of SO{n) on V m , 
the polynomials of homogeneous degree m, can be decomposed into 
irreducible subspaces as 



(41) V m = E m _ 2k ® C(W) fe , 

m-2fc>0 

where Ei are the spherical harmonic polynomials of degree i (cf [23J). 
This is a special case of the general Hua-Schmid decomposition (cf 
[2]). The following relation holds between the Fock-Fischer norm and 
the ^-norm on the space E m _ 2k <8> C(ww t ) k (cf [2j). 



(42) ||p 



|2 _ \\P\\,5F 



(f)m-fc(l' - 

for p G E m _2k ®C(ww t ) k . We have the following decomposition of J$? v 
under K: 

Proposition 7. (Faraut-Kordnyi, \2\) a) If v > then 
(43) ^|jc = ^ £ m _ 2fc ® C(^') fc , 

m-2fc>0 

where E m ^ 2 k is the space of spherical harmonic polynomials of degree 
m — 2k. Moreover, we have the following expansion of the kernel 
function: 



(44) h(z, w)~ v = # 

m-2A:>0 ' fc 



(m-k,fc)(2> W) 



where K( m -k,k) is the reproducing kernel for the subspace E m _ 2k <8> 
€.{zz t ) k with the Fock-Fischer norm. The series converges in norm 
and uniformly on compact sets of S) x S) . 
b) Ifu= then 

(45) ^k = 0X}^ 

We will later need the norm of izz 1 ) k in ML. 
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Proposition 8. 

(46) u«f)x= M ,' ( %» 

Proof. A straightforward computation shows that 

' + ■ ■ • + + ■ ■ ■ + z 2 n ) k = (2 2 k(k - 1) + n2k)(zf + ■■■ + zl) k ~ l 



dz\ ' ' dz 2 n 
Proceeding inductively, we obtain 

The Fock-Fischer norm is computed in the ^-coordinates Wi = y2zi, 
so 



and 



Hence 



d 2 d 2 \ " u ( d 2 d 



+ •■• + ■^-5-1 = 2" fc I --^ + • ■ ■ + 



dz\ dz 2 J \dwf dw 2 



>r\\% = k\ (f) 



.2J k 

and an application of Prop. [7] gives the result. □ 



5. Branching of ir u under the subgroup H 

5.1. A decomposition theorem. Recall the irreducible (projective) 
representations ir u from the previous section. Our main objective is to 
decompose these into irreducible representations under the subgroup 
H. The fact that is a totally real form is reflected in the restrictions 
of the representations k v to H. 

Proposition 9. The constant function 1 is in and is an L-invariant 
cyclic vector for the representation ix v : H — > ^/(J^ u ). 



20 



HENRIK SEPPANEN 



Proof. First note that 

K v (z,ho) = J h {hr x zy v i n K v {hr x z, 0)4(0)-/" 

= Jhioy^j^izy^K^z, o) 

= jJOp^Tr^/i)^^ 

Suppose now that the function / 6 is orthogonal to the linear 
span of the elements iz v (h)l, h G H. By the above identity we have 

f(h0) = (f,K u (;hO)) u 

= 0. 

Since H acts transitively on JT, / is zero on j?T. Hence it is identically 
zero. □ 

We want decompose the representation of H into a direct integral 
of irreducible representations. For the definition of a direct integral 
over a measurable field of Hilbert spaces we refer to Naimark ([17]). 
The following general decomposition theorem is stated in several ref- 
erences (e.g. pEH]), but the author has not been able to find a proof 
of it in the literature. A proof for abelian groups can be found in 
[T7] . The proof we present below is based on the Gelfand-Naimark 
representation theory for C*-algebras. 

Theorem 10. Let it be a unitary representation of the semisimple Lie 
group H on a Hilbert space, M' . Suppose further that L is a maximal 
compact subgroup and that the representation has a cyclic L-invariant 
vector. Then n can be decomposed as a multiplicity-free direct integral 
of irreducible representations, 

(47) vr= / n x dfi(\), 

J A 

where A is a subset of the set of positive definite spherical functions 
on H and for A 6 A, 7r A is the corresponding unitary spherical repre- 
sentation. 

Proof. We consider the Banach space L x (i/). This is a Banach *- 
algebra with multiplication defined as the convolution 

(f*g)(x)= / f(y)g(y' 1 x)dy 

Jh 
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and involution defined by 

Recall that the representation n extends to a representation of the 
Banach algebra L l (H) by 

/ >-> / f(x)n(x)dx. 
Jh 

We will also denote this mapping of L l (H) into £$(J$?) (the set of 
bounded linear operators on Jt?) by it. This representation will also 
be cyclic as the following argument shows. Denote by £ the L-invariant 
cyclic unit vector for H. Vectors of the form 

7T(f e )(7T(h 1 ^+---+7T(h n )0, 

where {f e } is an approximate identity on H, will then be dense in Jtf. 
Moreover the identity 

7r(/)(7r(/n)£ + ■ • • + n{K)t) = <{R h ~S + ■■■ + RkWZ, 

holds for / G L 1 ^!!) and hi,...,h n G H. (Here Rhf denotes the 
right-translation of the argument of /; / i— > f(-h). We similarly 
define L h f .) Hence vectors of the form 7r(/)£, where / G L l (H), form 
a dense subset in Jfc '. 

The function $ defined as 

(48) *:tt(/)'-M/)£,0 

extends to a state on the C*-algebra ^ generated by 7r(L 1 (if)) and 
the identity operator. It is a well-known fact from the theory of C*- 
algebras that the norm-decreasing positive functionals form a convex 
and weak*-compact set (cf [16]). For a C*-algebra with identity, the 
extreme points of this set are the pure states. Therefore, $ can be 
expressed as 

(49) $ = / <p x dn, 

Jx 

where X is the set of pure states and /i is a regular Borel measure 
on X (cf [22], Thm. 3.28). We recall the Gelfand-Naimark-Segal 
construction of a cyclic representation of a C*-algebra associated with 
a given state (cf [E]). In this duality, the irreducible representations 
correspond to the pure states. So each tp x in (T49l) parametrises an 
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irreducible representation of 7r(L 1 (if)) on some Hilbert space H x with 
a ^(L 1 (if)) -cyclic unit vector £ x . 

Herafter we will, by an abuse of notation, write $(/) for $(7r(/)) 
and correspondingly for the functionals y? x . 

We define a unitary operator T : Jff — > J x H x d\i that intertwines 
the actions of ^ by 

(50) V :;;(/)-• (/)(,•}• / /-'(//!• 

To see that this is well-defined, suppose that 7r(/)£ = 0. Then we 
have 

(51) = = ° 
i.e., 

(52) $(/* * /) = 
By ([49]) we have 

(53) $(/* * /) = / M/* * VX-C-).,-'//' = o. 

./if 

Therefore ir x (f)£ x = for almost every x and hence T is well defined 
on a dense set of vectors. Note that (1531 ) also shows that T is isometric 
on this set and it therefore extends to an isometry of Jt? into J x H x d\i. 

Consider now the subalgebra, L 1 (if)#, consisting of all /^-functions 
that are left- and right L-invariant, i.e., 

Uf = Rtf = f, 

for all I in L. This is a commutative Banach *-algebra (cf [6], Ch. IV). 
We know that (p x o tt : L^L 7 )* — ► C is a homomorphism of algebras 
and is therefore of the form (cf [6], Ch. IV) 

(54) <p w (f) = [ f(h)Mh)dh, f G L\H)*, 

where (fi x is a bounded spherical function. In fact, this formula holds 
for all /^-functions on H, as the following argument shows. 
Since £ is L-invariant, the identity 

7T(/)£ = 7T(i2,/)£ 

holds for all L 1 -functions / and I G L. Applying T to both sides of this 
equality (and using the fact that both L l (H) and L are separable), 



BRANCHING OF SOME HOLOMORPHIC REPRESENTATIONS OF SO(2,N)23 



we see that 

(55) -a. IX, = -AHiIK, 

holds for all / G L l (H) and I G L outside some set of measure zero 
with respect to /i. We now choose an approximation of the identity 
{r] e } on H, and by replacing it with {J L T] e (l ' l~ l )dl} if necessary, we 
may assume that it is invariant under the conjugate action of L. 
Consider now $ e defined by 

^(/) = <7r(/)7r(i7 6 )e,7r(i fc )0. 

We define the functionals ip X)e analogously for all x G X. Clearly 
$ e (/) — > $(/) as e — > and therefore 

limy> Xie (/) = 

holds for all /^-functions / outside some set of measure zero with 
respect to \x. (Again we use the separability of L 1 (if).) Using the 
L-conjugacy invariance of r) e and (1551 ). a simple calculation shows that 

<PxM) = ^,e(/ # ), 

where 

/W = / / f(hhl 2 )dhdl 2 , 

J L JL 

and by letting e tend to zero we get 

<Px{f) = fx{f # ) 

for almost every x. Hence 

V>M = I f(h)<f> x {h)dh, 
Jh 

for / G L\H). 

Since <p x also preserves the involution *, it is a positive linear func- 
tional, i.e., 

(56) / f(h)</> x (h)dh>Q, 

Jh 

for every / G L l (H), such that / = g * g*, for some g G L l (H). 

Lemma 11. Suppose that ip is a bounded spherical function such that 
f H f(h)<f(h)dh > for all f G ^{H) of the form f = g * g* for some 
g G L 1 (iJ). Then <p is positive definite. 
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= g * g* as in the statement, we have 



f(h)<p(h)dh= / / g(y)g(h-iy)d W (h)dh 
h Jh Jh 



g(y) / g(h l y)tp(h)dhdy 
h Jh 




g(y)g(z)v(yz l )dzdy 

' H JH 

Pick any complex numbers C\, . . . , c n and elements xi, . . . , x n in H and 
fix e > 0. We can choose a compact set K C H, containing all Xi in 
its interior, and a neighbourhood U of the identity such that 

(57) ^{xy-^-^x'y 1 - 1 ^ <e 

for all (x, y) and (x',y') in K x K such that yy'' 1 ) G U x U. 

We now choose disjoint neighbourhoods Ei of x« such that Ei C K 
and xj Ej, C Z7 for all z. Next, we choose an L 1 -function g in such a 
way that its support lies in [jEi and g has the constant value Ci/\Ei\ 
on Ei, where 



\Ei\ = / dh. 



Now we have 



/ / g{y)g{z)<p{yz X )dzdy = ^2 / 9{y)g{z)^p{yz V )dzdy 

(58) = Y^aiyd^^yj^mEjl 



1,3 



for some yi in Ei. The choice of g implies that the sum in (l58l ) equals 



%3 

This yields 



I / / g(y)g(z)ip(yz l )dzdy -y^CiCjCfixiX, 1 )] 
Jh Jh ~" 

< 2j Nlcjlly^^" 1 ) _ ^(^i^ -1 )! < ^ 2 sup |cj|e. 
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This shows that 

^CjCjtptxjXj 1 ) > 0, 

and hence (p is positive definite. □ 

Since every positive definite spherical function defines an irreducible, 
unitary, spherical representation of H, it also gives rise to a represen- 
tation L l (H). Its restriction to the subspace of L-invariant vectors, 
E x will be L 1 (if) # -invariant and one-dimensional (cf |6j, Ch. IV). If 
the state cp x corresponds to the spherical function (f> x , we denote by 
(tt x ,H x ) both the representations of H and of L}{H) that it induces. 
Corresponding to this cyclic representation of L l (H) with cyclic unit 
vector 4> x , we have that the state / i— > {7r x {f)4> x ,4> x )x is 

(iTx(f)<f>x,<l>x)x = / f(h)(ir x (h)(j) x ,(f) x ) x dh 
Jh 

f(h)(L h 4> x ,(f> x ) x dh 
= [ f{h)<p x (h- l )dh 

JH 

= [ f(h)JJh)dh. 

JH 

Therefore this representation of ^(H) is unitarily equivalent to the 
one given by the Gelfand-Naimark-Segal correspondence, i.e., we can 
regard the representation as coming from a representation of the group 
H. 

The operator T clearly intertwines the group representations tt and 
J x 7r x dfi. The only thing that remains is to prove that T is surjective. 
Suppose that c = {c x } is orthogonal to T(7r(L 1 (i/)), i.e., 

(~Af )C- <:,)., d fi = 0. 

We observe that the restriction of T to the space Jff L of L-invariant 
vectors intertwines the representations of 7r(L 1 (Lf) # ) on Jtf L and 
J x E x dn. The mapping 

7T(/) ^ (X ^ (fx(f)) 

is the Gelfand transform that realises the commutative C*-algebra 
generated by ir(L l (H)#) and the identity operator as the algebra, 




20 



HENRIK SEPPANEN 



C(X), of continuous functions on X. Continuous functions of the 
form ty(x) = </? x (/*) 5 where /* G are dense in C(X). For 

such \l> we have 

/ (n x (f)£ x ,C x ) x ^(x)dn = / (7T x (f * /*)&,, Cr)*^ 

= 

From this we can conclude that (using once more the separability of 
L l (H)#) for all a; outside a set of //-measure zero, the equality 

(n x {f)£x,c x ) x = 

holds for all / G L l (H)#. Since the vectors £ x are L 1 (iJ) # -cyclic, we 
can conclude that c = and this finishes the proof. □ 

Remark. The measure \x in the above theorem is called the Plancherel 
measure for the representation ir. 

5.2. Extension and expansion of the spherical functions. Con- 
sider the mapping R : ,W V — > C°°(3£) defined by 

(Rf)(x) = h(x,x) v/2 f(x),x G % 

(see [28]). When v > n — 1, R is in fact an if -intertwining operator 
onto a dense subspace of L 2 (^T, dt) (where di is the if -invariant mea- 
sure on J2T) and the principal series representation gives the desired 
decomposition of n u into irreducible spherical representations. This is 
a heuristic motivation for studying the functions R~ X ip\, where tp\ is 
a spherical function on . 



Theorem 12. Let v > (n — 2)/2. The function R 1 <f\(z) is holomor- 
phic on *3) and has the power series expansion 

R~ l (fx{z) = ^PkWekiz), 

k 

where e^{z) is the normalisation of the function z \— > (zz') fc in the 
J^ v -norm, and the coefficients Pfc(A) are polynomials of degree 2k of X 
and satisfy the orthogonality relation a) If v > then 
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1 

2^ 

r 



r(i + «A)r(^i + ix)v(u -z=± + i\) 



n 



r 1/ 



n- 2 



T(2iA) 

r(i/)<y w - 



p l /,fc(A)p i ,,/(A)dA 



b)Ifv< then 



r(± + «A)r(^i + iA)r(i/ - ^ + a) 



T(2zA) 



+ 



1 

27 

r(t/)r(z/ - g^)r(n - i - z/)r(f - ^ 

r(n- 1 - 2u) 
n — 1 



/7 



r,-in„ 



2 

71-2 



n — 1 



r(z/)4z- 



Proof. Recall the root space decomposition for h. Let (, ) denote the 
inner product on ac that is dual to the restriction of the Killing form 
to a. Let a denote a/ (a, a). 

In this setting the spherical function (p\ is determined by the formula 
(cf |B], Ch. IV, exercise 8) 



(59) p A (exp(t&)0) = 2 F 1 (a / , 6', c'; - sinh(a(t£ e )) 2 ), 
where 

1 f 1 / ' \ A 1 /n-1 
° = 2 \ 2 ma + m2a + ' °' / = 2 ["2 h 

1 Z^ 1 A I fn-l 

2 V 2 m ° + m2a ~ ' ' a °' = 2 I ~~2 

1/1 \ 1 /n+1 

c = - I -m Q + m 2 « + 1 

Letting x = exp(t£ e )0 = tanhi, (l59l takes the form 

(60) ^ x ) = aFl ^ 1 y t ^-^- i ) 

1 — XX 1 



2n 
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<p x (x) = 2 Fi(o', b\ d; — ) = (1 - xxY 2 F 1 {a', d - b', d- xx l ) 



By Euler's formula (cf [5J) we have 

1 — XX 

For the function R~ x if\ we thus get the expression 
(61) Rr^xiz) = (1 - zz*)~ v+af 2 Fi(a', d - b', c; zz l 
Expanding (IBTi ) into a power series yields 



oo m 



m=0 Z=0 v ' K * 

noticing that \zz t \ < 1 for zG f. Next, we use the following simple 
identities: 

/ f\ (v-a')m [y-o!) m 

(u-a) m -i ^ 



(m-l)\ 



(u-a' + (m- l))t {-l) l (-(u -a' + m- 1)), 
m! 



(m-Z + l)r 
Substitution of these in (l62l) yields 

(63) ^Va(^) 

= (y ~ aQm y> (a')i(d ~ , t\m 

^ ml f-< (c'U-iv - a> + m - l))d ] 

m=0 1=0 

The inner sum in (1631) can be recognised as a hypergeometric function, 
i.e., we have 

E(a')i(d — b')i(—m)i , , . 

; ,w T iyT = 3*2 a', c' - b', -m; d, -{v - d + m - 1 ; 1 . 

i=o {d)i{-{v-a> + m-l)) l 

Now we use Thomae's transformation rule (cf [5]) for the function 3 F 2 : 

3 F 2 (a' , c — b' , — m; c', — (z/ — a' + m — 1); 1) 
(-(u-a' + m-l)-(d-b')) m 



{-(v -a' + m- l)) m 
x 3*2 (c — a', c' — 6', — m; 1 + (d — b') + {v — a' + m — 1) — m; 1) 
We finally obtain the following expression: 



R l <p\(z) = ^c„^ fe (A)( 



zzY, 



k=0 
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where 



[v-*=2) k n/ , 1 + iA 1-iA n n-2 



c n ,u,k( X ) = j£j 3F 2 (-k, , g ' v 2~> X ) 

Recall the continuous dual Hahn polynomials (cf |27j) 

(64) S k (x 2 ; a, b, c) = (a + %(a + c) k 

x 3-^2( — k, a + ix, a — ix; a + b, a + c; 1) 
We can thus write 

« W) = E ( . )t( „ _ U, )tH a (( 5 ft a ■ — ■ - — ) («■) 

(z2;*) fc 



(zz t ) k \ 



k=0 

For the orthogonality relation in the claim, we refer to [27]. □ 

5.3. Principal and complementary series representations. In 

this section we let \i (=£ti/) be the finite measure on the real line that 
orthogonalises the coefficients p k (X) in ( 1591 ). Let A„ be its support. As 
we saw above, \i can, depending on the value of v, either be absolutely 
continuous with respect to Lebesgue measure or have a point mass at 
A = i{y — {n — l)/2), i.e., we either have 

K = (0, oo) \J{i(u - (n - l)/2)}, ue((n- 2)/2, (n - l)/2) 

or 

A u = (0,oo), u>(n- l)/2. 

We will now construct explicit realisations for the spherical represen- 
tations nx corresponding to the points A G A u on Hilbert spaces H\. 
For A in the continuous part in A, the underlying space H\ will be 
L 2 (S n ~ v ) and for the discrete point i{y — {n — l)/2), H\ will be a 
Sobolev space. 

We will hereafter suppress the index v and simply denote the support 
of \i by A. 

Lemma 13. If g e H , then g transforms the surface measure, o, on 
S n - X as 

da(gQ = J g (()~ da(r)). 



30 



HENRIK SEPPANEN 



Proof. Clearly it suffices to prove the statement for automorphisms of 
the form 

g = exp£ v ,v G W l . 

Moreover we can assume that ( = e±, since any ( G S 11 ^ 1 can be 
written as le±, where I G L, and 

exp^(/ei) = (exp£J)(ei) = (ir 1 exp^J)(e 1 ) = (Zc^-i(exp £„))(ei) 

= Zexp (Arf(/ _1 )^)(ei) = Zexpf ,-i„(ei). 

Consider now the tangent space of R n at e\. We have an orthogonal 
decomposition 



T ei (R n )=T ei (S 



n— 1 x 



'1- 



At ge\ we have the corresponding decomposition 

T gei (W ri )=T gei (S n - 1 )®Rge 1 . 
Since H preserves 

^( ei )T ei (5"- 1 )=T sei (^ 1 ), 

and by completing e\ and gei to orthonormal bases for their respective 
tangent spaces, dg{e\) corresponds to a matrix of the form 

/ c \ 

* * * 
v * * * 
\ * * * J 

Hence 

(65) J fl (ei) = cJpi^^ei), 
where 

(66) c = (d#(ei)ei,#ei). 
We next determine this constant c. 



We have 



c = (d#(ei)ei,0ei) = lim(d#(rei)rei, grex). 

r— >1 



For fixed r < 1 we have 

exp£„(rei) = tt + B{u, u)^ 2 B(rei, — u) -1 (rei + Q(rei)u) 
= u + dg(rei)(rei + Q(rei)u) : 
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and 

where it = tanhf. Since Q{re\)u = 2(it, rei)rei — it, we get 

(68) (dflr(rei)rei,flr(rei)) 

= (1 + 2 (it, rei))\dg(rei)rei\ 2 + (d^(re 1 )re 1 , u — dg{re\)u) 

For any z G ^f)M n and v,w E R n , the identity 

(69) (c^K «,) = ki °£ 9 z *\ v, dg{z)- l w) 

can be established using the transformation properties of the function 
h and the operator B. Applying (1691) in the cases z = rei, i> = rei, 
and w = dg(rei)rei and 
w = u — dg(rei)u, repectively, yields 

(70) (a^(T-ei)rei,d^(rei)rei) = r r 

and 

(71) (dg(rex)rex,u- dg(rex)u) 

%(rei),#(rei)) x 

= T7 x ( re i ; d#(rei) u-u). 

n,{rei,rei) 

The expressions above and an elementary computation shows that 
(j68|) can be written as 

(72) {dg{rex)re l: g{re x )) 



1 + 2 (it, rei] 


) + 


li 


2 


1 - 


it 


2 



h(rei, rei) 

By the transformation rule for the Bergman kernel 

h(g(rei),g(re x )) = \J g (rei)\ 2/n h(rei,rei). 

So, 

c = lim \JJrei)\ 2/n r 2 
i — »i 



1 + 2(u ) rei] 


) + 


It 


2 


1 - 


it 


2 



h{u,u) L i z 
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Comparing with the expression (l67t ). we have determined the constant 

c = Ue,) 1 '^ 

and this finishes the proof. □ 

For A in the continuous part of A, the corresponding representation 
is a principal series representation described by the following propo- 
sition. (We will hereafter follow Helgason and in this context denote 
S 1 ™ -1 by B. The measure a will be denoted by db.) 

Proposition 14. For any real number \, the map h i— > T\(h), where 

T X (h)f(b) = J h - 1 (b) 1 ^ £ f(h- 1 b) 

defines a unitary representation of H on L 2 (B). 
Proof. We have 

/ \J h -i{b)^\ 2 \f(hr l b)\ 2 db = [ J h -,{hb) 2 -t\f{b)\ 2 d{hb) 

Jb Jb 

= [ J h {b)-^\f{b)\ 2 J h {b)^db 
Jb 

= [ \f(b)\ 2 db, 
Jb 

where the last equality follows by lemma [131 □ 

It is well known that the representations T\ above are unitarily 
equivalent to the canonical spherical representations associated with 
the corresponding functionals A on ac (cf [11], ch. 7). 

In order to realise the representation r A for A = %{y— {n — l)/2), we 
consider the following Hilbert spaces. 

Definition 15. For ^ < a < let be the Hilbert space 

completion of the C°°-functions on S n ~ l with respect to the norm 

11/11%=/ / f(OW)K((,vTdo-(Odo-( V ) 

Using the action of H on S" 1-1 , we can define a unitary representa- 
tion of H on of the form 

o a : f^J h -A-ff(h- x -),heH, 
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where (3 = —a + (n — l)/n. The unitarity follows from 

J h -AQ P f(h-\)J h -^yf(h-^)K{C^Yd<y(Qda( V ) 



n — 1 ./ Cn-1 



n — l / Cn-1 



J h (()-e- a+ ^J h (v)^- a+ ^f(0f(v)K((,vrdcr(0da(r,). 



n— 1 / cn-1 



In fact, this representation is irreducible (cf pQ). We denote this repre- 
sentation by a a . One can prove that for a = u/n and A = v— (n— 1)/2, 
a a and t x are unitarily equivalent. 

Recall the expression in Cor. [6] for the spherical functions. In this 
setting we write it as 



B 



(fx(x) = I e x , b (x)db 

where 

e\,b{x) 



1/2 \ iX+ P 



h(x, x\ 
h(x, b) 

by Cor. [H For fixed z G 3 1 and A G A, i? -1 ^^^) is a function 
in L 2 (B). Moreover, n u (H) makes sense as a group of mappings on 
0(@), the set of holomorphic functions on @. We have a relationship 
between these representations. 

Lemma 16. For every g G H and A £ A, 

(73) n u {g)T X {g)R- l e Kb {z) = BT l e Xtb (z). 
Correspondingly, for X G h, we have the relation 

(74) 7r i ,(X) J R- 1 e A , fe (z) = -r A (X) J R- 1 e Aife (z). 

The proof is straightforward by applying the transformation rules 
for the function h(z,w). 

5.4. The Fourier-Helgason transform. The purpose of this section 
is to construct an if -intertwining unitary operator between the Hilbert 
spaces J#l and J A H x dfi. 

Any holomorphic function, /, on @ has a power series expansion 



(75) /(*) = £/«, 
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where f a = ald * a (0). We can collect the powers of equal homogeneous 
degree together and write 

(76) /(*) = £/*(*), 

k 

where is of homogeneous degree k. We now consider the mapping 

(■,■)„ :VxO(®)^C 

defined as 

(77) (f,g)u = Y,(f>9*)»- 

k 

Observe that the definition makes sense since every polynomial is or- 
thogonal to all but finitely many g k . 

Definition 17. If f is a polynomial in Ml, its generalised Fourier- 
Helgason transform is the function / on A x B defined by 

(78) f(\b) = (f,RT 1 e XJb ) v 

Proposition 18. (i) If the polynomial f is in M U L , then f is L- 
invariant and 

\\f\\l= [ WfWldn, 

J A 

where || • \\\ is the norm on H\, and the Fourier- Helg as on transform 

extends to an isometry from Mj" onto L 2 (A,dfi). 

(ii) The inversion formula for L-invariant polynomials 

(79) f{z) = [ /(A)iTVA(^(A) 

J A 

holds. Moreover, the above formula holds for arbitrary L-invariant 
functions, when restricted to the submanifold 3£ . 

Proof. Writing 

R^e\,b = ^2 Ca ^' = S 6x ' b ' k 

a k 

and 

a k 
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we see that the coefficients and polynomials of homogeneous degree k 
are related by 

(80) c a (X)= [ c a (X,b)db 

J B 

and 

(81) p k {X)e k (z) = / e x ,b,k( z ) db 

Jb 

respectively. Therefore we have 

k 

= / K u (l)fdl,e x ^ k ) v 

k ^ L 

k ^ L 

= / ir x (l)e x ,b,kdl)v 

k ^ L 

= (f,R~ Va)„- 
This proves the L-invariance. Moreover, we have 

(f,R~ 1 Vx ) u = J2Pk~W(f^k)u- 

k 

Hence 

f WfWldn = X)K/,e*>,| 2 = ll/U*. 

This proves the first part of the claim. 

To prove the inversion formula, we now let / be an L-invariant 
polynomial and a; be a point in 3tf)~R n . Since we have an estimate of 
the form 

(82) I^VaWI < (l-\x\ 2 )-^C(x), 

where C is some function of x, independently of A, the integral 

/ /(A)irVA(^(A) 

J A 
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makes sense for real x. We then have 



/(A) j R-Va(^)^(A) = J2 f (/,e,)^ fe (A)i2-VA^)^(A) 

= ^2(f,ek)v / ^Pfc(A)p j (A)e i (x)d^(A) 



A 3 



= f{x). 

Now let / G J$f u L be arbitrary. We choose a sequence of polynomials 
/„ G J4? U L such that 

/ = lim/ n . 

Since the evaluation functionals are continuous, we have 
f(x) = lim f n (x) = lim / f n (X)R' 1 (p x (x)dfj J {\) 

n— >oo n— >oo 

for every real point x. By Jensen's inequality and (1821) 

2 

(/(A) - /n(A))irVA(aO^(A) 
<MA) / 17(A)- M\)\ 2 C(x)(l-\x\ 2 y»dn(\). 

J A 

Hence 

/(*) = f f(X)R- 1 Vx (x)d f ,(X). 



Thus the inversion formula holds for real points, x. To see that the 
formula holds for arbitrary points when / is a polynomial, we note 
that both the left hand- and the right hand side of the formula define 
holomorphic functions on @. Since they agree on the totally real form 
JT, they are equal. □ 

Theorem 19 (The Plancherel Theorem). For v > (n — 2)/2, the 
Fourier- Helg as on transform is a unitary isomorphism from the H- 
modules J^f u onto the H -module f A H\d/j, i.e., 

Mh)fj(\,b)=T X (h)f(\,b), 

for h G H , and 

[dfi. 



2 _ / II f ||2. 
v 
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Proof. We divide the proof into three steps: 

(i) We prove that the Fourier-Helgason transform intertwines the ac- 
tion of the Lie algebra of H. 

(ii) We use (i) to prove that the norm is preserved. 

(Hi) We conclude that the group actions are intertwined from (i) and 
(ii). 

We will see that these properties actually imply that the Fourier- 
Helgason transform is surjective. 

Consider now the corresponding representations of the Lie algebra 
f). These will also be denoted by n u and T\ respectively. Moreover 
they extend naturally to representations of the universal enveloping 
algebra, 11(f)), of f). 

Let I 6 I). If / is a polynomial in J%,, then differentiation of the 
mapping 

*- J cxpt x(zT /n f((exptX)z) 
at t = shows that 7i u (X)f is also a polynomial, and 

nUX)f(X,b) = ^MX)/,e AAfc >, 
k 

= ^2(f,-^u(X)e XAk ) u 

k 

= J2(f,rx(X)e XAk ) v 

k 

= r A (X)(/, J R- 1 e A , fe ) 1 „ 

which proves (i). 

To prove the second step, we recall that the adjoint representation 
of L on f) extends to an action on 11(f)) as homomorphisms of an asso- 
ciative algebra. The L-invariant elements in 11(f)) form a subalgebra, 
H(f)) i . We let p denote the projection 

x^y Ad(i)xdi 

of 11(f)) onto lt(f)) L . This action of L connects the representations of 
H and 11(f)) according to the following identity: 

TT^KPO^" 1 ) = 7T,(Ad(l)X) : 
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for I G L and X G 11(f)). 

Since the vector 1 G J4? u is cyclic for the representation of H, it is 
also cyclic for the representation of IX(f)). Hence it suffices to prove 
that the norm is preserved for elements of the form n u (X)l, where 
X G 11(f)). In the following equalities, we temporarily let r denote the 
direct integral of the representations T\, and analogously we let (, ) 
denote the direct integral of the corresponding inner products. 

(ir„(X)l,ir v (X)l) v = (ir v (Xyir v (X)l,l) v 
= (-n v (X*)l,l) v 

Since the vector 1 is L-invariant, the last expression equals (—ir u (p(X 2 ))l, 1) 
and by proposition (fl8l) . we have 

(-7r,(p(X 2 ))l,l),= (-7r,(K^))l,l). 

By (i), the expression on the right-hand side equals (— r(p(X 2 ))l, 1), 
and since 1 is L-invariant, we have 

(-r(p(X 2 ))l ) l) = (-r(X 2 )l,l). 

Thus (ii) is proved. 

To prove (Hi), we recall the following equalities (on the respective 
dense spaces of analytic vectors): 

7r„(exp(X)) = e w "W 
r A (exp(X)) = e^W. 

From this and the facts that H is connected and that the Fourier- 
Helgason transform is bounded operator, we immediately see that (Hi) 
holds. 

To see that the operator is surjective, note that by (ii) and (Hi) 
Mf)l,l) u = [ (r x l(X,-),l(X,-)) x dfi, 

J A 

for / G L 1 (if)*, i.e., we can write the positive functional 

f^(n u (f)l,l) u 

as an integral of pure states with respect to some measure. By unique- 
ness, it is the measure in Theorem [101 Since the Fourier-Helgason 
transform intertwines the group action, it is the intertwining operator 
constructed in Theorem ITOl Thus it is surjective. □ 
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Theorem 20 (The Inversion Formula). If f is a polynomial in 
then 

(83) f(z)= [ [ 7(A,6)i?- 1 e Ai6 (^)d6^(A). 

J A JB 

Proof. Take h G H . Define 

U{z)= J 7r v (l)7r u (h)f(z)dl 

This is a radial function, and we have that 

(84) fi(0) = Jh-m-f(h- 1 0). 
Prop. [18] gives 

(85) A(0) = / A(A)^-Va(^)^(A). 

J A 

Moreover 

fi(\) = (fi,R- 1 <Px) v = {( 'Ku{l)^u{h)fdl i R- x <p x ) v 



= (<K„(h)f,R- l ip x ) 

(86) 

By Thm. [19] we have 

K(/i)/,irVA), = {fiK v {hr l )R- l ip x ) u 



(/, / ^(h-^R-'ex^db), 

JB 

(f, [ n^h-^R-'ex^db), 

JB 

(f, [ T X (h)Br 1 e X j>db) v 

JB 

(f, [ J h -r{b)^R- l e x , h -^db) v . 

JB 



The integrand above has a power series expansion where the coeffi- 
cients are functions of b. If we integrate, we obtain a holomorphic 
functions for which the coefficients in the power series expansion are 
obtained by integrating the aforementioned coefficients over B. Hence 
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we can proceed as follows. 

(f, / Jh-^(b) 1 ^ £ R~ 1 ex,h-^bdb) u = / Jh-i{b)~ ! i? Ji {f ) R~ 1 e\ )h -i b ) v db 

JB J B 

= I J h -i(b)^J{A,h~ l b)db 

JB 

= [ J h -l{hb)=^J(\,b)J h (b)^db 
JB 

(87) = / f(X,b)J h {b)^db. 

JB 

It is easy to see that 

(88) Jh{b)^ = A-i(0)^- 1 e Ai6 (/i- 1 0), 
and so combining (1531) . (1551) and (1571) finally yields 

(89) /(^ 1 0)= / / j{\b)R' l ex, b {h^)dbd^. 

J A JB 

Thus the inversion formula holds for real points, hence for all points 
by the same argument as in the proof of Prop. HH1 □ 



6. Realisation of the discrete part of the decomposition 

We recall the earlier defined complementary series representations. 
The following theorem states that a v / n is the representation corre- 
sponding to the singular point in the decomposition theorem. 

Theorem 21. The operator T v defined by the formula 
(TJ)(z)= [ f(OK u (z,C)da(0 

JS"- 1 

is a unitary H -intertwining operator from %,/ n onto an irreducible 
H-submodule of 

Proof. First of all we note that T v maps functions in %,/ n to holomor- 
phic functions on @ and thus ir u has a meaning on the range of T v . 
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We start by showing that T u is formally intertwining. We have 
T v {a v/n )f{z) = [ J h -i{Q-»! n+ ^f{h- l QK v (zX)da{Q 

j h (cr /n -^f(c)Mz,hc)j h (o^da(o 

-i 

J h {O v/n f{OK v {h- l zX)J h {h- l z)-"nJ h {C)^da{C) 

-i 

J h -i(z)5 [ f(C)K l/ (h- 1 z,C)da(0, 



i.e., 



The next step is to prove that the constant function 1 is mapped into 
Ml and that its norm is preserved. Note that for a = u/n, K(z, () a = 
K v (z,£), and by Prop. [7] we have an expansion 

m—k,k) 

m-2k>0 

where the coefficients c m ^{y) are given explicitly. Now, since K u ((, ei) 
is SO(n — l)-invariant and the action of SO(n — 1) is linear, each 
K( m ,k)((, ei) must also be SO(n— l)-invariant. Hence, K( m) ty(£, e-y) can 
be assumed to be </> m __2fc(C)(CC*) fc ? where 4> m -2k is the unique element 
in E m _2k that assumes the value 1 in e±. Therefore 

(90) / K(C, V ) a da(0 

= [ K u ((,l ei )dl [ K u {r 1 C,e 1 )dl 

J L J L 

(91) = Yl c ™^) / ( rl C(^ 1 C) t ) fc m - 2fc (r 1 c)rf/ 

m-2fc>0 ^ L 



(92) 



c ^)U l ) k I <p m -2k{l^Odl 



m-2k>0 



Since SO(n) acts irreducibly on E m _ 2 k and the function J L 4> m -2k(l~~ lz )dl 
is an SO(n) -invariant element in E m _ 2 k it must be identically zero un- 
less m — 2k = 0. Since 

Pill, = [ I k v (c,v)MOMv), 
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the computation above implies that 

llillL = / E c ^(^)(cc') fc ^(0 

J ° k=0 



°° ( v) h (v-^) h 



k=0 k=0 11 v ' 11 & 



ZZ l ) k \ 2 



00 A n-2 



(93) = £ 



^Jfcl^ — Jfc 



fc=0 V 2^« 



On the other hand, the equalities (I90l) - (l92l) also show that 



x < v ) k (v-^) k , L 



fc=0 V 2^ fc 



If we compare f|93l) and (l9lj) . we see that T v l E and that ||1||%? , 
llllL. Recall that 



m 

and that the representation of f) on the algebraic sum ® m i? m (S' n ~ 1 ) 
is irreducible. Hence 

E^- 1 ) = Span c {<7„/„(Xi) . . . ^ /n (X fc )l|X,, e f), 1 < i < k] 
in 

Since T v interwines the representations of f), we have that n u is an 
irreducible representation of f) on the space T v (@ m i? m (5' n_1 )) C J^ v . 
By Schur's lemma ( [13J , ch.4) 

(T u f,T u g) u = c(f,g)<jf v/n , 

for some real constant c. Putting, / and g equal to the constant 
function 1 and applying, we see that c = 1. Therefore, T u extends to 
a unitary operator 



rn 

and we have proved the theorem. □ 
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7. Realisation of the minimal representation 7T( n _ 2 )/2 

In this section we show that the representation 7T( n _ 2 )/2 of H is 
irreducible by realising it as a complementary series representation. 

We recall the space ^ v / n from the previous section and the corre- 
sponding operator T v . 

Theorem 22. T(„_ 2 )/2 is a unitary H -intertwining operator from ^( n _2)/2 

Onto J£( n -2)/2- 

Proof. Recall that 



(95) V(n-2)/n = E^S- 1 ) 

rn 

and that the sum is a decomposition into S , 0(ra)-irreducible subspaces. 
If we let V(n-2)/n denote the set of all finite sums in (|95l) . <Jt n -2)/n 
defines a representation of [ on P( n _ 2 )/ n . The polynomial (Ci + ^C2) m is 
a highest weight vector in E m for this representation. Moreover, the 
power series expansion of K( n _ 2 )/ n shows that T( n _ 2 )/ 2 is a polynomial 
in E m . Since T( n _ 2 )/ 2 intertwines the [-actions, T( n _2)/ 2 ((Ci + zC 2 ) m ) is 
a highest weight vector space for 7T( n _ 2 )/ 2 ([), i- e -) 

(96) (T (n _ 2)/2 (Ci + iC 2 ) m ) W = C m (z + iz) m , 

for some constant C m . We now determine C m . Choose z = w\(l, —i, 0, . 
where w is a complex number with \w\ < 1. In this case zz l = 
0, {z + iz) m = w m . We now compute (T (n _ 2 )/ 2 ((Ci + «'C 2 ) m )(z)- 

K (n -2)/ n (z,()({i+i{ 3 ) m 

(1 - w(Ci - < 2 ))- (n - 2)/n (Ci + < 2 ) m ^(C) 



It— 1 



This integral only depends on the first two coordinates and can hence 
be converted to an integral over the unit disk, U (cf [2T] Prop 1.4.4). 

(1 - w(d - «Ca))- ( "- 2,/B (Ci + K*) m M0 

-1 

rC— r 

-kr / (i-^C)- (n - 2)/ri C m (i-iCI 2 ) (n - 4)/2 rfm(C). 
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We have the power series expansion 



(i-K)-^ 2)/2 = f;(^) (<) k 

k=0 ^ ' k 

Recall that (1 — w()~ n ^ 2 is the reproducing kernel for the weighted 
Bergman space J^f n /2{U), defined as 

M> n/2 {u) = {/ e 0(U) | ^ |/(C)| 2 (i - |CI 2 ) (n - 4)/2 ^(C) < oo}, 

Polynomials of different degree are orthogonal in Jt? n / 2 (U) and hence 
we have 

/ (1 - w 0- {n - 2)/n C m (1 - \C\ 2 ) (n ^ )/2 dm(C) 
Ju 

= f E (^) ( z O k r(i - \c\ 2 ) (n - 4)/2 dm(o 

Ju k=0 \ * J k 

= IJL i^r) (l) m w m c(i - ici 2 ) ( "- 4)/2 ^(o 



k=0 
7lW m , 



where the last equality follows from the reproducing property in J^ n /2{U)- 
Summing up, we have 

77 — 2 

(97) (T (n _ 2)/2 (d + < 2 n w = -^-(z 1 + tz 2 r 

From this and the intertwining of the [-action, it follows that 

(98) T (n _ 2)/2 ( E m {S n - 1 ) j C E m 

To compute the norm of T( n _ 2 )/ 2 (p) where p G E^S 71-1 ), we first fix 
r < 1 and consider the polynomial T( n _ 2 y 2 (p(rz)). By definition 



T {n _ 2)/2 {p){rz) = / K v (rz,Op(OMQ 
(99) = / K v (z,rQp(Qdv(Q. 
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The norm is given by 

\\T (n _ 2)/2 (p)(r-)\\l= [ [ p(OW)KM:rv)da(C)da(rj). 

J S"- 1 Js"- 1 

Finally, we let r — > 1 and obtain 

\\T (n -2),2(p)\\l = [ [ P (C)MMC:V)d<j(C)da(r l ). 

From this and the orthogonality of the spaces E^, it follows that 
T( n - 2 )/2 maps (0 m -E m (5' n ~ 1 ) isometrically onto (0 m -B m )- Hence it 
extends to a unitary operator from c to( n -2)/2n onto J$?( n -2)/2- D 
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